Introduction to Time Series Analysis - 04

Introduction to Time Series Analysis - 04

This note is for course MATH 545 at McGill University.

Lecture 10 - Lecture 12

---

Estimation of mean $\mu$ and auto-covariance function $\rho(h)$ for stationary $\{X_t\}$ when $E[X_t]=\mu$

$\hat{\mu}=\bar{X}_{n}=\frac{X_{1}+\cdots+X_{n}}{n}$

$E\left[\bar{X}_{n}\right]=\frac{E\left[X_{1}\right]+\cdots+E\left[X_{n}\right]}{n}=\mu$

$E\left[\left(\bar{X}_{n}-\mu\right)^{2}\right]=\operatorname{MSE}\left(\bar{X}_{n}\right)=\operatorname{Var}\left(\bar{X}_{n}\right)$

$\begin{aligned} \operatorname{Var}\left(\bar{X}_{n}\right) &=\operatorname{Var}\left(\frac{1}{n}\left[X_{1}+\cdots +X_{n}\right]\right) \\ &=\frac{1}{n^{2}} \text{Var} \left(X_{1}+\cdots+X_{n}\right) \\ &=\frac{1}{n^{2}} \sum_{i=1}^{n} \sum_{j=1}^{n}\text{Cov}\left(X_{i}, X_{j}\right) \\ &=\frac{1}{h^{2}} \sum_{i=1}^{n} \sum_{j=1}^{n} \gamma(i-j) \\ &=\frac{1}{n^{2}} \sum_{h=-n}^{n}(n-|h|) \gamma(h) \\ &=\frac{1}{n} \sum_{h=-n}^{n}\left(1-\frac{|h|}{n}\right) \gamma(h) \end{aligned}$

(Note that here $(n-|h|)$ is the number of possible observed lags. And we have $0\leq |i-j| \leq n$ which means $n+1$ values of $|i-j|$)

If $\gamma(h) \rightarrow 0$ as $n \rightarrow \infty$, then $\operatorname{Var}\left(\bar{X}_{n}\right)=\operatorname{MSE}\left(\bar{X}_{n}\right) \rightarrow$ as $n \rightarrow \infty$

If $\sum_{h=-\infty}^{\infty}|\gamma(h)|<\infty$, then $\lim _{n \rightarrow \infty} n \operatorname{Var}\left(\bar{X}_{n}\right)=\sum_{n=-\infty}^{\infty}|\gamma(h)|$

If $\{X_t\}$ are also Gaussian, then $\bar{X}_{n} \sim N\left(\mu, \frac{1}{n} \sum_{|h|<\infty}\left(1-\frac{|h|}{n}\right) \gamma(h)\right)$

To do testing and identical estimation $\bar{X}_{n} \pm Z_{Q / 2} \frac{\sqrt{\Gamma}}{\sqrt{n}}$, where $\Gamma = \sum_{|h| < \infty} \gamma(h)$

Plugin $\hat{\Gamma}=\sum_{|h|}\hat{\gamma}(h)$

Example: AR(1) process

Let $\{X_t\}$ be defined by $X_t - \mu = \phi(X_{t-1}-\mu) + Z_t$ where $|\phi| < 1$ and $\{Z_t\}\sim WN(0, \sigma^2)$

So, $\begin{aligned} \Gamma &=\sum_{|h| < \infty} \gamma(h) = \sum_{|h| < \infty} \frac{\sigma^2 \phi^{|h|}}{1-\phi^2} \\ &=(1+2\sum^{\infty}_{h=1} \phi^{|h|})\frac{\sigma^2}{1-\phi^2} \\ &=(1+2\frac{\phi}{1-\phi})\frac{\sigma^2}{1-\phi^2} \\ &=\frac{1-\phi+2\phi}{1-\phi}\frac{\sigma^2}{(1+\phi)(1-\phi)} \\ &=\frac{\sigma^2}{(1-\phi)^2} \end{aligned}$

So 95% Confidence Interval for $\mu$:

$\bar{X}_n \pm \frac{1.96}{\sqrt{n}}\frac{\sigma}{\sqrt{(1-\phi)^2}} = \bar{X}_n \pm \frac{1.96}{\sqrt{n}}\frac{\sigma}{|1-\phi|}$

$\hat{\gamma}(h)=\frac{1}{n}\sum_{t=1}^{n-|h|}(X_{t+|h|}-\bar{X}_n)(X_t-\bar{X}_n)$

$\hat{\rho}(h)=\frac{\hat{\gamma}(h)}{\hat{\gamma}(0)}$

Both biased in finite samples, but consistent

$\hat{\Gamma}_k = \left [ \begin{array}{cccc} \begin{array}{l} \hat{\gamma}(0)\\ \hat{\gamma}(1) \end{array} & \cdots & \begin{array}{l}\hat{\gamma}(k-1) \\ \hat{\gamma}(k-2) \end{array} \\ \vdots & \ddots & \vdots\\ \begin{array}{l} \hat{\gamma}(k-1) \end{array} & \cdots & \begin{array}{l} \hat{\gamma}(0) \\ \end{array} \end{array} \right ]$

For all $k<n$, $\hat{\Gamma}_k$ will be non-negative definite. Not obvious how to estimate for $k\geq n$, even for $k$ close to $n$, $\hat{\Gamma}$ will be unstable.

(Jenkin’s rule & theorems: need $n=50$ and $h\leq \frac{n}{4}$)

In large samples without large lags, can approximate the distribution of $(\hat{\rho}(1), \cdots, \hat{\rho}(k))$ by:

$\hat{\rho} \sim MVN(\rho, \frac{1}{n}W)$

where $W$ is a $k\times k$ covariance matrix with elements computed by a simplification of butlet’s formula:

$w_{ij}=\sum^\infty_{k=1}\{\rho(k+i)+\rho(k-i)-2\rho(k)\rho(i)\}\times\{\rho(k+j)+\rho(k-j)-2\rho(k)\rho(j)\}$

Example

$\{X_t\}$ iid $\Rightarrow$ $\rho(0)=1$ and $\rho(h)=0 \quad |h|>1$

So $w_{ij}=\sum^\infty_{k=1}\rho(k-i)\rho(k-j)$

We have $w_{ii}=1$ and $w_{ij}=0$, so $\hat{\rho}(h) \sim N(0, \frac{1}{n})$ as $W=I$

We have an MA(1) process with $X_t=Z_t+\theta Z_{t-1}$ where $Z_t \sim WN(0, \sigma^2)$

Then $X_t=\sum^\infty_{k=0}\psi_k Z_{t-k}$ and

$\Rightarrow \psi_k = \begin{cases} 1, \text{for } k=0 \\ \theta, \text{for } k=1 \\ 0, \text{for } k\neq\{0, 1\} \end{cases}$

So $\gamma_X(h)=\sum^\infty_{j=-\infty} \psi_j \psi_{j-h}\sigma^2 = \begin{cases}(1+\theta^2)\sigma^2, \text{for } h=0 \\ \theta\sigma^2, \text{for } h=1 \\ 0, \text{for } h\geq 2 \end{cases}$

So $\rho(0) = 1$ and $\rho(\pm 1) = \frac{\gamma(1)}{\gamma(0)} = \frac{\theta}{1+\theta^2}$

From the formula $w_{ij}=\sum^\infty_{k=1}\{\rho(k+i)+\rho(k-i)-2\rho(k)\rho(i)\}\times\{\rho(k+j)+\rho(k-j)-2\rho(k)\rho(j)\}$, we have for $i=j$

$w_{ii}=\sum^\infty_{k=1}\{\rho(k+i)+\rho(k-i)-2\rho(k)\rho(i)\}^2$

$w_{11}=(\rho(o)-2\rho(1))^2 + \rho(1)^2 = 1-3\rho(1)^2+4\rho(1)^4$

If it is the case of MA(1), $\hat{\rho}(1) \sim N(\frac{\theta}{1+\theta^2}, \frac{1}{n}(1-3\rho(1)^2+4\rho(1)^4))$

For $i>1$, $w_{ii}=\sum^\infty_{k=1}\{\rho(k+i)+\rho(k-i)-2\rho(k)\rho(i)\}^2 \\=\rho(0)^2 + \rho(1)^2 + \rho(-1)^2 = 1+2\rho(1)^2$

(Note: because in this case, $\rho(k+i)$ is always 0, $\rho(k)\rho(i)$ is always 0, $\rho(k-i)$ is nonzero only when $k = i, i+1, i-1$)

Prediction (Forecasting)

Goal: Find linear construction of $X_1, \cdots, X_n$ that forecasts $X_{n+h}$ with minimum MSE

Assume best linear predictor of $X_{n+h}$ is:

$P_nX_{n+h} = a_0+a_1X_n+a_2X_{n-1}+\cdots+a_nX_1$

We want to find $a_0, a_1, \cdots, a_n$ to minimize:

$S(a_0, a_1, \cdots, a_n) = E[(X_{n+h} - (a_0+a_1X_n+a_2X_{n-1}+\cdots+a_nX_1))^2]$

$S$ is quadratic, bounded below by zero, and there exist at least one solution to $\frac{\partial}{\partial a_j}S(a_0, \cdots, a_n)=0$ for $j = 0, 1, \cdots, n$

By taking derivatives, it gives (assuming interchange $\frac{\partial}{\partial a_j}$ and $E[\cdot]$ safely)

[1]\begin{equation}\frac{\partial}{\partial a_0}S(a_0, \cdots, a_n)=0 \Rightarrow E[X_{n+h} - a_0 - \sum^n_{i=1}a_iX_{n+1-i}]=0 \end{equation}

[2]\begin{equation}\frac{\partial}{\partial a_j}S(a_0, \cdots, a_n)=0 \Rightarrow E[(X_{n+h} - a_0 - \sum^n_{i=1}a_iX_{n+1-i})X_{n+1-j}]=0 \end{equation}

From equation 1 we have:

$a_0 = E(X_{n+h}) - \sum_{i=1}^n a_i E(X_{n+1-i}) \\=\mu-\mu \sum^n_{i=1} a_i$ if $\{X_t\}$ is stationary

Plug it into equation 2, we have:

$E[(X_{n+h} - \mu(1-\sum^n_{i=1} a_i) - \sum^n_{i=1}a_iX_{n+1-i})X_{n+1-j}] = 0$

$E[(X_{n+h}-\mu)X_{n+1-j}]=E[\sum^n_{i=1}a_i(X_{n+1-i}-\mu)X_{n+1-j}]$

$\gamma(h+j-1) = \sum^n_{i=1}a_i E[(X_{n+h}-\mu)X_{n+1-j}] = \sum^n_{i=1}a_i \gamma(i-j)$

We have a matrix form of this formula $\Gamma_n\underset{\sim}{a_n} = \underset{\sim}{\gamma_n}(h)$ where $(\Gamma_n)_{ij} = \gamma(i-j)$

Therefore $P_nX_{n+h} = \mu + \sum^n_{i=1}a_i(X_{n+1-i}-\mu)$ where $\underset{\sim}{a_n}$ satisfies $\Gamma_n\underset{\sim}{a_n} = \underset{\sim}{\gamma_n}(h)$.

Obviously, $E[X_{n+h} - (\mu + \sum^n_{i=1}a_i(X_{n+1-i}-\mu))] = 0$

\begin{align}MSE &=E((X_{n+h} - P_nX_{n+h})^2) \\&=E(X_{n+h} - (\mu + \sum^n_{i=1}a_i(X_{n+1-i}-\mu))^2) \\&=E[((X_{n+h} - \mu) -( \sum^n_{i=1}a_i(X_{n+1-i}-\mu)))^2] \\&=E((X_{n+h} - \mu)^2) -2E((X_{n+h} - \mu)(\sum^n_{i=1}a_i(X_{n+1-i}-\mu))) + E((\sum^n_{i=1}a_i(X_{n+1-i}-\mu))^2) \\&=\gamma(0) - 2\sum^n_{i=1} E((X_{n+h}-\mu)(X_{n+1-i}-\mu))+\sum^n_{i=1}\sum^n_{j=1}a_iE((X_{n+1-i}-\mu)(X_{n+1-j}-\mu))a_j \\&=\gamma(0) - w\sum^n_{i=1}a_i\gamma(h+i-1) + \sum^n_{i=1}\sum^n_{j=1}a_i\gamma(i-j)a_j \\&=\gamma(0) - w\sum^n_{i=1}a_i\gamma(h+i-1) + \sum^n_{i=1}a_i(\sum^n_{j=1}\gamma(i-j)a_j) \\&=\gamma(0) - 2\underset{\sim}{a_n}^T\underset{\sim}{\gamma_n}(h) + \underset{\sim}{a_n}^T\Gamma_n\underset{\sim}{a_n} \\&=\gamma(0) - \underset{\sim}{a_n}^T\underset{\sim}{\gamma_n}(h) \end{align}

(Note: because $\underset{\sim}{a_n}$ satisfies $\Gamma_n\underset{\sim}{a_n} = \underset{\sim}{\gamma_n}(h)$, so $\underset{\sim}{a_n}^T\underset{\sim}{\gamma_n}(h) = \underset{\sim}{a_n}^T\Gamma_n\underset{\sim}{a_n}$ )

Example AR(1)

$X_t = \phi X_{t-1} + Z_t$, where $|\phi|<1$ and $\{Z_t\} \sim WN(0, \sigma^2)$

We try $a_1=\phi, a_k = 0$ for $k = 2, \cdots, n$

A solution that works is $\underset{\sim}{a_n} = (\phi, 0, \cdots, 0)$ and $P_nX_{n+1} = \underset{\sim}{a_n}^T\underset{\sim}{X_n} = \phi X_n$

$E((X_{n+1} - P_nX_{n+1})^2) = \gamma(0) - \underset{\sim}{a_n}^T \underset{\sim}{\gamma}(1) \\=\frac{\sigma^2}{1-\phi^2} - \phi\gamma(1) =\frac{\sigma^2}{1-\phi^2} - \phi\frac{\sigma^2\phi}{1-\phi^2} \\= \sigma^2$

Now let $Y$ and $W_1, \cdots, W_n$ be any random variabe with finite second moments and means $\mu = E(Y)$ and $\mu_i = E(W_i)$, and covariance $Cov(Y, Y), Cov(Y, W_i), Cov(W_i, W-j)$

Let $\underset{\sim}{W} = (W_n, \cdots, W_1)$, $\underset{\sim}{\mu} = (\mu_n, \cdots, \mu_1)$, and $\underset{\sim}{\gamma} = Cov(Y, \underset{\sim}{W})$, $\Gamma = Cov(\underset{\sim}{W}, \underset{\sim}{W})$ such that $\Gamma_{ij} = Cov(W_{n+1-i}, W_{n+1-j})$

By exactly the same methods from before, we show that the best linear predictor of $Y$ given $W$ is:

$P(Y|W) = \mu_Y + \underset{\sim}{a}^T(\underset{\sim}{W} - \underset{\sim}{\mu_W})$

where $\underset{\sim}{a} = (a_1, \cdots, a_n)$ is any solution of $\Gamma\underset{\sim}{a} = \underset{\sim}{\gamma}$

Return to AR(1), assume that we observe $X_1$ and $X_3$ but not $X_2$

Let $Y=X_2$ and $W=(X_1, X_3)$, we have

$\Gamma = \begin{pmatrix} \frac{\sigma^2}{1-\phi^2} & \frac{\phi^2\sigma^2}{1-\phi^2}\\ \frac{\phi^2\sigma^2}{1-\phi^2} & \frac{\sigma^2}{1-\phi^2} \end{pmatrix}$

$\gamma = \begin{pmatrix} Cov(X_2, X_1) \\ Cov(X_2, X_3) \end{pmatrix} = \begin{pmatrix} \frac{\sigma^2}{1-\phi^2}\phi \\ \frac{\sigma^2}{1-\phi^2}\phi \end{pmatrix}$

As $\Gamma\underset{\sim}{a} = \underset{\sim}{\gamma} \Rightarrow \begin{pmatrix} 1 & \phi^2\\ \phi^2 & 1 \end{pmatrix}\underset{\sim}{a} = \begin{pmatrix} \phi\\ \phi \end{pmatrix}$

So $\underset{\sim}{a} = \begin{pmatrix} \frac{\phi}{1+\phi^2}\\ \frac{\phi}{1+\phi^2} \end{pmatrix}$

$P(X_2|X_1, X_3) = \frac{\phi}{1+\phi^2}X_1 + \frac{\phi}{1+\phi^2}X_3 = \frac{\phi}{1+\phi^2}(X_1+X_3) + \frac{\phi}{1+\phi^2}\cdot 0$

$P(\cdot|W)$ is a prediction operator and it has useful properties:

Let $E(U^2) < \infty, E(V^2) < \infty, \Gamma=Cov(\underset{\sim}{W}, \underset{\sim}{W})$

Let $\beta, a_1, \cdots, a_n$ be constants

1. $P(U|W) = E(U) + a^T(W - E(\underset{\sim}{W}))$ where \Gamma_\underset{\sim}{a} = Cov(U, \underset{\sim}{W})
2. $E((U-P(U|\underset{\sim}{W}))\underset{\sim}{W}) = \underset{\sim}{0}$ and $E(U-P(U|\underset{\sim}{W}))=0$
3. $E((U-P(U|\underset{\sim}{W}))^2) = Var(U) - \underset{\sim}{a}^TCov(U, \underset{\sim}{W})$
4. $P(a_iU+a_2V+\beta|W) = a_1P(U|W) + a_2P(V|W)+\beta$
5. $P(\sum_{i=1}^na_iw_i + \beta|\underset{\sim}{W}) = \sum_{i=1}^na_iw_i+\beta$
6. $P(U|\underset{\sim}{W}) = E(U)$ if $Cov(U,\underset{\sim}{W})=0$
7. $P(U|\underset{\sim}{W})=P(P(U|\underset{\sim}{W},\underset{\sim}{V})|\underset{\sim}{W})$

(Note: for property 7, $P(U|\underset{\sim}{W}, \underset{\sim}{V})=\mu_U+a_W(W-\mu_W)+a_V(V-\mu_V)$, $P(U|\underset{\sim}{W}) = P(\mu_U + a_W(W-\mu_W)+a_V(V-\mu_V)|W)$)

Assume $\{X_t\}$ is a stationary process with mean 0 and auto-covariance function $\gamma(\cdot)$, we can solve for $\underset{\sim}{a}$ to determine $P_nX_{n+h}$ in terms of $\{X_n, \cdots, X_1\}$.

However, for large $n$, inventory $\Gamma$ is not fun!

Perhaps we can use linearity of $P_n$ to do recursive prediction of $P_{n+1}X_{n+h}$ from $P_nX_{n+1}$.

If $\Gamma_n$ id non-singular, then $P_nX_{n+1} = \underset{\sim}{\phi^T_n}\underset{\sim}{X} = \phi_1X_n + \cdots + \phi_nX_1$